Scene independent method for image formation in lenslet array imagers

ABSTRACT

A method and system is provided for performing high-resolution image assembly regardless of observed scene content. An imaging system, including a focal plane array and lenslet array can be calibrated to account for subimage shifts. A calibration module can determine the subimage shifts by calculating an average point source position reference point coordinates for each of the subimages, and then determining the difference between the average point source position and the reference point coordinates for each subimage. The imaging system can then be calibrated utilizing the subimage shifts for each of the plurality of subimages. Finally, an assembly module can perform a high-resolution image assembly with the calibrated imaging system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to provisional patent applicationentitled, “Scene Independent Method for Image Formation in Lenslet ArrayImagers,” filed on Jul. 18, 2008, and assigned U.S. Application No.61/081,845; the entire contents of which are hereby incorporated byreference.

FIELD OF THE INVENTION

The invention relates generally to flat imaging systems. Moreparticularly, the invention relates to a method or system that canperforms high-resolution image assembly from undersampled sub-imagesobtained from compact imaging system using lenslet array

BACKGROUND

In a conventional system a traditional imager system typically consistsof a single lens and a focal plane array. Recent demand for highlycompact thin imaging systems led to broad investigation efforts insensor design based on focal plane array technologies. A system based onthis more recent architecture for reducing the imager thickness can be asystem that uses a lenslet array to accumulate a series of sub-images. Aspecific example of this new architecture is the thin observation moduleby bound optics (TOMBO) system.

This TOMBO architecture differs from the conventional system because itis a flat architecture that can consist of a lenslet array, a bafflepreventing optical cross-talk, and an equivalent focal plane array. Ifthe lenslet array is to be equal to the conventional system lens thenn×n number of lenslets can be required to cover the area of thedetector, ideally keeping the number of incident photons the same forboth configurations. As a result, the depth of optics and sub-image sizeare n times smaller than the conventional system. The sub-images of thelenslet array can be processed in a manner similar to multi-framesuper-resolution to obtain a fully sampled, high-resolution image. Theprocessing task is to reconstruct an image that has resolution as closeas possible to the conventional camera lens and the same focal planearray.

The principle of multi-frame super-resolution is that the pixel valuesfrom all sub-images are mapped onto an upsampled image plane. Eachsub-image position on the super-resolved image plane is determined byits global shift with respect to a reference subimage. In multi-framesuper-resolution processing these shifts can be estimated either byiterative procedure or by sub-image registration. The former method isunsuitable for real time applications due to possible convergenceproblems and heavy computational load, which dramatically increases withrespect to the number of sub-images. The accuracy of the latter approachis highly dependent on the nature of the observed scene, such as havingsome knowledge of the probability density function or noisecharacteristics of the observed scene.

Accordingly, there remains a need for a method or system that canperforms high-resolution image assembly from undersampled sub-imagesobtained from compact imaging system using lenslet array utilizingscene-independent processing and in a method suitable for real-timeapplications.

SUMMARY OF THE INVENTION

The present invention provides a method for performing high-resolutionimage assembly regardless of observed scene content by measuring aplurality of point source positions for a plurality of subimages. Thepoint source positions can be projected from a point source, such as alaser, onto a small number of pixels within each subimage, and can beimaged. The point source can then be shifted multiple times to create asub-pixel shift of the point source position within each of thesubimages, and to create multiple point source positions for eachsubimage. An average point source position for each of the subimages canbe determined by averaging the plurality of point source positionsobtained after each point source shift. A centroid for each of theplurality of subimages can be determined. The centroid of one subimagecan be utilized to produce a reference point coordinate. A subimageshift for each subimage can then be determined by taking the differencebetween an average point source position for each subimage and thereference point coordinate of the subimage chosen as the reference. Theimaging system can then be calibrated utilizing the subimage shifts foreach of the plurality of subimages and a high-resolution image assemblycan be performed with the calibrated imaging system.

The present invention provides a system for performing high-resolutionimage assembly in a lenslet array system regardless of observed scenecontent. The system includes an imaging system that includes a focalplane array and a lenslet array. Furthermore, a calibration module canbe included that is configured to measure a plurality of point sourcepositions for a plurality of subimages, determine a centroid for each ofthe plurality of subimages, determine a subimage shift for each of theplurality of subimages, and calibrate the imaging system utilizing thesubimage shifts for each of the plurality of subimages. Finally, anassembly module can be included that is configured to perform ahigh-resolution image assembly with the calibrated imaging system

The present invention provides a method for calibrating an imagingsystem regardless of observed scene content to perform high-resolutionimage assembly by defining a plurality of subimage shifts, the subimageshifts varying based on distance between an object and a target, for aplurality of distances less than a far-field calibration. The subimageshifts can be calculated using a formula δ=xF/D where δ is the subimageshift, D is the distance, F is a lenslet focal length, and x is aseparation of a lenslet center from an overall array center. An imagingsystem can be calibrated by utilizing the plurality of subimage shifts.The imaging system can then perform a high-resolution image assemblywith the calibrated imaging system. The imaging system can determine thedistance from the object to the target; select the subimage shiftassociated with that distance; and process the image with the selectedsubimage shift. In one embodiment, the imaging system can determine thedistance from the object to the target by receiving an input from a userof the imaging system comprising the distance from the object to thetarget. In another embodiment, the imaging system can determine thedistance from the object to the target by empirically deriving thedistance from the object to the target.

These and other aspects, objects, and features of the present inventionwill become apparent from the following detailed description of theexemplary embodiments, read in conjunction with, and reference to, theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging system in accordance with anexemplary embodiment of the invention.

FIG. 2 is a scatter plot illustrating the subimage shifts of a pluralityof subimages.

FIG. 3 is a flow chart illustrating an exemplary method for performinghigh-resolution image assembly regardless of observed scene content.

FIG. 4 is a flow chart illustrating an exemplary method for measuring aplurality of point source positions for a plurality of subimages.

FIG. 5 is a block diagram of an imaging system in accordance with analternative exemplary embodiment of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Referring now to the drawings, in which like numerals represent likeelements, aspects of the exemplary embodiments will be described inconnection with the drawing set.

FIG. 1 is a block diagram of an imaging system 100 in accordance with anexemplary embodiment of the invention. Recent demand for highly compactflat imaging systems that use existing focal plane array (FPA)technologies has led to innovative new sensor designs. One specificdesign, which can be used in accordance with an exemplary embodiment ofthe invention, for reducing the imager thickness is the thin observationmodule by bound optics (TOMBO) system 110. The TOMBO system 110 can usea lenslet array 140 to accumulate a series of sub-images of a target150. The TOMBO system 110 comprises a lenslet array 140, an optical mask(or baffle) 130 that can prevent optical cross talk, and an equivalentfocal plane array 120.

In the TOMBO system 110, the lenslet array 140 can comprise a pluralityof individual lenslets. Typically, the lenslet array is arranged in ann×n lenslet structure. For example, the lenslet array 140 can be an 8×8lenslet array. The TOMBO system 110 can operate by obtaining a pluralityof low-resolution subimages collected by the lenslet array 140, and thenassembling the subimages into a high-resolution image. In the example ofan 8×8 lenslet array, the TOMBO system 110 can collect 64 low-resolutionsubimages.

In a lenslet array 140, typically all individual lenslets are fixed; andtherefore, the relation between the sub-images should remain constant.This constant relation can allow for a single precise measurement ofeach sub-image position to determine a subimage shift and successive useof the subimages and shift in high-resolution image restoration. Thesesubimage shifts can be measured precisely in a controlled laboratoryenvironment, and then can be incorporated into the image assemblyalgorithm. This a priori calibration can provide constant and nearlymaximum possible image resolution enhancement (relative to undersampledsub-images) for all successive imager data collections independent ofviewing conditions or scene content. Image assembly algorithm is thusdramatically reduced in complexity and computational time compared toall existing image assembly algorithms.

In accordance with an exemplary embodiment of the invention, the imagingsystem 100 can comprise a calibration module 160 that can be configuredto determine the subimage shifts. The calibration module 160 can beconfigured to measure a plurality of point source positions for aplurality of subimages. The calibration module 160 can image a series ofthe point source positions and the point source positions can beprojected from a point source, such as a laser, onto a small number ofpixels within each subimage. For example, in an exemplary embodiment,the laser can be pointed at the lenslet array 140. This can map as apoint spread function onto a small number of focal plane array pixels ineach subimage.

To minimize the subsampling effect of the relatively large size ofpixels, the point source can be shifted to create a sub-pixel shift ofthe point source position within each of the subimages. For example, thepoint source can be shifted relative to the imager in the planeperpendicular to the virtual target-imager line such that each stepmotion corresponds to a sub-pixel shift of the point source on the imageplane. This step of shifting the point source to produce new pointsource positions with sub-pixel shifts can be performed multiple timesto create multiple point source positions for each subimage. Forexample, at laser position a, subimage-1 through subimage-n (in an 8×8array, n=64) will each have a point source position. When the laser isshifted to position b, subimage-1 through subimage-n will each have adifferent point source position.

The calibration module 160 can store the multiple point source positionsat different point source starting positions for each subimage. Aftercollecting the multiple point source positions, the calibration module160 can determine an average point source position for each of thesubimages by averaging the multiple point source positions obtainedafter each point source shift. The average point source position foreach subimage can subsequently be used in determining the subimageshifts.

For each subimage on the focal plane array 120 there can be a centroid.The centroid positions can be utilized by the calibration module 160 todetermine subpixel shift for each of the plurality of subimages. Forexample, the centroid of one subimage can be used as a reference pointcoordinate. The resulting reference point coordinate can be utilized bythe calibration module 160 to determine the subimage shift for eachsubimage. To calculate the subpixel shift for the remaining subimages,the calibration module 160 can determine the difference between theaverage point source position for each subimage and the reference pointcoordinate of the subimage chosen as the reference. These differencescan represent the subimage pixel shifts of each of the subimages and isillustrated, as an example, in FIG. 2.

FIG. 2 is a scatter plot illustrating the subimage shifts of a pluralityof subimages. The numbers in the scatter plot in FIG. 2 represent theparticular numbered subimage (e.g. subimage-1 (210) through subimage-64(220)). The x and y planes represent the relative shifts of eachsubimage.

After determining the relative shifts for each subimage, the calibrationmodule 160 can incorporate the subimage shift for each of the pluralityof subimages into an assembly algorithm in the assembly module 170. Theassembly module can be included in the imaging system 100. For example,if the point x=0 and y=0 is considered the reference point coordinate,then in order to assemble a high-resolution image, subimage-1, where thex-coordinate is ˜0.8 and the y-coordinate ˜0, needs to be shifted 0.8pixel in the negative x-direction and 0 pixel in y-direction.

The assembly module 170 can be configured to perform a high-resolutionimage assembly with the imaging system 110. The purpose of the imagereconstruction process is to combine all low-resolution sub-images intosingle high-resolution image. The assembly module can utilize theassembly algorithm to shift subimages from an irregular grid onto aregular upsampled grid, and then combine the shifted subimages on theregular upsampled grid into a single high-resolution image. For thehigh-resolution image assembly, interpolation methods, known to one ofordinary skill in the art, can be chosen such as nearest neighbor,linear, cubic, or higher order, if necessary.

The assembly module 170 can then filter the upsampled image containingall the information incorporated in the subimages using the Wienerfilter. The Wiener filter can provide correction for optical system blurand detector sampling effects. This operation utilizes a priori knowninformation about an optical system, such as lenslet point spreadfunction and detector shape; therefore avoiding more complex iterativeprocessing which attempts to infer lenslet characteristics.

In an exemplary embodiment of the present invention, the calibrationmodule 160 and assembly module 170 can be implemented in a computersystem that comprises instructions stored in a machine-readable mediumand a processor that executes the instructions.

FIG. 3 is a flow chart illustrating an exemplary method for performinghigh-resolution image assembly regardless of observed scene content. InStep 310, a plurality of point source positions can be measured for aplurality of subimages. In an exemplary embodiment of the presentinvention, a calibration module 160 can be configured to measure theplurality of point source positions for the plurality of subimages. Forexample, a point source, such as a laser, can be pointed at the lensletarray 140 to produce a plurality of point source positions, which arerepresented as a small number of pixels within each subimage. This canmap as a point spread function onto a small number of focal plane arraypixels in each subimage.

FIG. 4 is a flow chart illustrating an exemplary method for measuring aplurality of point source positions for a plurality of subimages. InStep 410, a series of the point source positions within each subimagecan be imaged. This step can be accomplished, for example, by thecalibration module 160. Subsequently, in Step 420, to minimize thesubsampling effect of the relatively large size of pixels, the pointsource can be shifted to create a sub-pixel shift of the point sourceposition within each of the subimages. This step of shifting the pointsource to produce new point source positions with sub-pixel shifts canbe performed one or more additional times to create multiple pointsource positions for each subimage. The calibration module 160 can storethe multiple point source positions at different point source startingpositions for each subimage.

In step 430, after collecting the multiple point source positions, thecalibration module 160 can determine an average point source positionfor each of the subimages by averaging the multiple point sourcepositions obtained after each point source shift. The average pointsource position for each subimage can subsequently be used indetermining the subimage shifts.

Returning to FIG. 3, in Step 320, a centroid for each of the pluralityof subimages can be determined. The centroid positions can be utilizedby the calibration module 160 to determine subpixel shift for each ofthe plurality of subimages in Step 330. For example, the centroid of onesubimage can be used as a reference point coordinate. To calculate thesubpixel shift for the remaining subimages, the calibration module 160can determine the difference between the average point source positionfor each subimage and the reference point coordinate of the subimagechosen as the reference.

In Step 340, an imaging system that utilizes the subimage shifts foreach of the plurality of subimages can be calibrated by incorporatingthe subimage shift for each of the plurality of subimages into anassembly algorithm in an assembly module 170. In Step 350, ahigh-resolution image assembly can be performed with the calibratedimaging system. For example, if the point x=0 and y=0 is considered thereference point, then in order to assemble high-resolution imagesubimage 1 (x-coordinate is ˜0.8 and y-coordinate ˜0) needs to beshifted 0.8 pixel in the negative x-direction and 0 pixel iny-direction.

FIG. 5 is a block diagram of an imaging system 500 illustrating analternative exemplary embodiment of the invention. As previouslydescribed, a lenslet array imager can be configured to produce an arrayof images of a scene. The images can be arranged in a fixed pattern asdetermined by the geometry of the system, particularly the opticalproperties and geometrical spacing of the lenslet arrays. Combining theimages into a so-called “super” image involves knowing the subimageshift to build up a super-sampled conglomerate image built from some orall of the individual images. A method and system for calculating asubimage shift, calibrating an image system, and performing ahigh-resolution image assembly with the calibrated imaging system isdiscussed previously.

However, in an alternative exemplary embodiment, whenever objects in thescene are close enough, the subimage shifts begin to noticeably deviatefrom the far-field calibration data. This situation is illustrated inFIG. 5, where an object 510 is a distance D from the lenslet arrayimager 520, which is made up of multiple lenslets. Rays traced from theobject 510 typically strike each lenslet of the lenslet array imager 520at an angle θ that depends on D. Thus, the image of the object as seenby each lenslet will shift an amount δ from the center of the lenslet byan amount dependent on D, the lenslet focal length F, and the separationx of the lenslet center from the overall array center. Assuming paraxialray tracing, known to one of ordinary skill in the art, the followingformula for δ can be used:δ=xF/D

Accordingly, a method for accounting for the proximity shifting of theimage features at close distances can be accomplished. In one exemplaryembodiment of the present invention, a dependence of the shift asdescribed above can be assumed, and a measured shift matrix, or subimageshift, can be modified each time an image is assembled. For example, thelenslet array 520 can be calibrated in the far field, e.g., where D isso much larger than F that for all x in the lenslet array, δ is verysmall compared to a pixel on the detector array. An assembly algorithmcould then modify the calibrated subimage shifts whenever an actualimaged scene has objects closer to the sensor, e.g., where D isrelatively small.

In an alternative exemplary embodiment of the present invention, amethod for accounting for the proximity shifting of the image featurescan be provided by defining a set of calibrated shift matrices, orsubimage shifts, with a different shift matrix, or subimage shift, foreach of a several different D's, including a far-field and anearest-focus calibration. For example, the calibrated system could havemultiple settings for different D values. The imaging system could thenallow a user to choose the appropriate shift matrix for a given image tobe used in the assembly algorithm. Alternatively, the assembly algorithmcould reconstruct the imagery with several of the calibration matricesand then select the best result (either by an assembly algorithm or bypresenting them to a human user). In one embodiment, the set of subimageshifts, or matrices, can be stored in a calibration module 160.

The imaging system can be calibrated with a plurality of subimage shiftsbased on different distances from the imaging system to a target. Afar-field setting, which is calibrated in accordance with FIGS. 1-4, canbe applicable for all distances over a certain distance. For exampleonly, the far-field distance could be used for all distances over tenfeet. However, for closer distances (e.g., less than ten feet), theimaging system can have multiple additional subimage shift settings. Byway of example, the imaging system could have one setting for picturestaken at a distance of five feet, and another setting for pictures takena distance of one foot, as well as many other settings for distancesless than the far-field setting. The settings for different distancescan then be incorporated into the assembly algorithm to be utilized inprocessing the subsequent image.

In an alternative exemplary embodiment of the present invention, anothermethod for accounting for the proximity shifting of the image featurescan be provided that combines the two previously described methods. Inthis embodiment, the imaging system can use a shift dependence tointerpolate between the measured shift matrices. For example, if thereis a shift matrix for images taken of a scene at different distances D₁and D₂, and the current data is of a scene at a distance D₃, withD₁<D₃<D₂, then the shift dependence formula given above could be used tomodify the shift matrix at a distance D₁ (or D₂) to be appropriate forD₃. Alternatively, since the multiple shift matrices are measured atknown object distances, a shift dependence formula δ(D) could beempirically derived for the system, and then used by the assemblyalgorithm.

For any of the discussed methods for accounting for the proximityshifting of the image features at a close distance, the distance D isnot necessarily known. In fact, D may vary throughout the image, withsome objects in the foreground and some in the background. In practice,either a user or an algorithm (or a combination) might employ shiftmatrices from multiple distances and compare the resulting reconstructedimages, selecting the best one. An algorithm might also be able toextrapolate ranges from by triangulating known feature sizes in theimagery. In another method, the imaging system might use a range finderor other independent measurement system to directly measure D to one ormore features in the scene.

For example, these settings can be located on the imaging system wherethe user can manually select her distance, and that information isincorporated into the assembly algorithm to process the subsequentimage. In an alternative embodiment, the user can proceed with capturingan image, and the assembling algorithm can process the image at multipledistances. The processed images can then be presented to the user toselect the most appropriate image.

The invention can comprise a computer program that embodies thefunctions described herein and illustrated in the appended flow charts.However, it should be apparent that there could be many different waysof implementing the invention in computer programming, and the inventionshould not be construed as limited to any one set of computer programinstructions. Further, a skilled programmer would be able to write sucha computer program to implement an exemplary embodiment based on theflow charts and associated description in the application text.Therefore, disclosure of a particular set of program code instructionsis not considered necessary for an adequate understanding of how to makeand use the invention. The inventive functionality of the claimedcomputer program will be explained in more detail in the followingdescription read in conjunction with the figures illustrating theprogram flow.

It should be understood that the foregoing relates only to illustrativeembodiments of the present invention, and that numerous changes may bemade therein without departing from the scope and spirit of theinvention as defined by the following claims.

The invention claimed is:
 1. A method for performing high-resolutionimage assembly, comprising the steps of: measuring a plurality of pointsource positions for a plurality of subimages; determining a centroidfor each of the plurality of subimages; determining a subimage shift foreach of the plurality of subimages; calibrating an imaging systemutilizing the subimage shifts for each of the plurality of subimages;and performing a high-resolution image assembly with the calibratedimaging system, wherein the step of performing the high-resolution imageassembly does not factor observed scene content.
 2. The method of claim1, wherein the step of measuring a plurality of point source positionsfor a plurality of subimages comprises imaging a series of the pointsource positions, wherein the point source positions are projected froma point source onto a small number of pixels within each subimage. 3.The method of claim 2, wherein the step of projecting the point sourcepositions onto the small number of pixels within each subimage isperformed with a laser.
 4. The method of claim 2, wherein the step ofimaging a series of the point source positions further comprisesshifting the point source to create a sub-pixel shift of the pointsource position within each of the subimages.
 5. The method of claim 4,wherein the step of shifting the point source to create a sub-pixelshift of the point source position within each of the subimages can beperformed a plurality of times to create a plurality of point sourcepositions for each subimage.
 6. The method of claim 5, comprising thestep of determining an average point source position for each of thesubimages by averaging the plurality of point source positions obtainedafter each point source shift.
 7. The method of claim 1, wherein thestep of determining a subimage shift for each of the plurality ofsubimages comprises the steps of: utilizing the centroid of one subimageto produce a reference point coordinate; and determining the subimageshift for each subimage as a difference between an average point sourceposition for each subimage and the reference point coordinate of thesubimage chosen as the reference.
 8. The method of claim 1, wherein thestep of calibrating an imaging system utilizing the subimage shiftscomprises the step of incorporating the subimage shift for each of theplurality of subimages into an assembly algorithm.
 9. The method ofclaim 1, wherein the step of performing a high-resolution image assemblywith the calibrated imaging system comprises the steps of: utilizing anassembly algorithm to shift subimages from an irregular grid onto aregular upsampled grid; and combining the shifted subimages on theregular upsampled grid into a single high-resolution image.
 10. Themethod of claim 1, further comprising the steps of: for a plurality ofdistances less than a far-field calibration, defining a set of subimageshifts, the subimage shifts varying based on distance between an objectand a target; and calibrating the imaging system utilizing the setsubimage shifts based on distance.
 11. A system for performinghigh-resolution image assembly in a lenslet array system, comprising: animaging system, comprising: a focal plane array; and a lenslet array; acalibration module configured to measure a plurality of point sourcepositions for a plurality of subimages, determine a centroid for each ofthe plurality of subimages, determine a subimage shift for each of theplurality of subimages, and calibrate the imaging system utilizing thesubimage shifts for each of the plurality of subimages; and an assemblymodule configured to perform a high-resolution image assembly with thecalibrated imaging system, wherein the assembly module configured toperform the high-resolution image assembly does not factor observedscene content.
 12. A method for calibrating an imaging system to performhigh-resolution image assembly, comprising the steps of: for a pluralityof distances less than a far-field calibration, defining a plurality ofsubimage shifts, the subimage shifts varying based on distance betweenan object and a target, calibrating the imaging system utilizing theplurality of subimage shifts; and performing a high-resolution imageassembly with the calibrated imaging system, wherein the step ofperforming the high-resolution image assembly does not factor observedscene content.
 13. The method of claim 12, wherein the step of defininga plurality of subimage shifts comprises the step of calculating asubimage shift for each of a plurality of distances less than afar-field calibration using the formula δ=xF/D, wherein δ is thesubimage shift, D is the distance, F is a lenslet focal length, and x isa separation of a lenslet center from an overall array center.
 14. Themethod of claim 12, wherein the step of calibrating the imaging systemutilizing the plurality of subimage shifts comprises storing theplurality of subimage shift values in a calibration module.
 15. Themethod of claim 12, wherein the step of performing a high-resolutionimage assembly with the calibrated imaging system comprises the stepsof: determining the distance from the object to the target; selectingthe subimage shift associated with that distance; and processing animage with the selected subimage shift.
 16. The method of claim 15,wherein the step of determining the distance from the object to thetarget comprises the step of receiving an input from a user of theimaging system comprising the distance from the object to the target.17. The method of claim 15, wherein the step of determining the distancefrom the object to the target comprises the step of empirically derivingthe distance from the object to the target.
 18. The method of claim 15,further comprises the steps of: processing the image with a plurality ofsubimage shifts; and presenting a plurality of images associated withthe plurality of subimage shifts.